Skip to content

Conversation

@BryanFauble
Copy link
Contributor

@BryanFauble BryanFauble commented Sep 4, 2025

Problem:

  • There is an updated helm chart available for the ocean kuberenetes controller
  • New nodes that were coming up were not joining the EKS cluster

Solution:

  • Use the following to find the new chart version, and generate a new values.yaml file:
-- Update all repos I have added:
helm repo update


-- Lists the repos I have added
helm repo list

-- Find the updated chart number
helm search repo spot

-- Generate a new values file I can copy/paste into the existing values.yaml
helm show values spot/ocean-kubernetes-controller > new_values.yaml
  • Update the user_data section of the spot terraform module to ensure that it will bootstrap correctly

https://awslabs.github.io/amazon-eks-ami/nodeadm/
https://registry.terraform.io/providers/spotinst/spotinst/latest/docs/resources/ocean_aws#user_data-1

Testing:

  • This has deployed to the sandbox k8s cluster
  • Nodes are joining the cluster updated to 1.33
image

@spacelift-int-sagebionetworks spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-kubernetes-deployments September 4, 2025 17:13 Inactive
drop:
- ALL

command: []
Copy link
Contributor

@linglp linglp Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to set command: []? Does it mean to use the default entry point?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not exactly sure. Many of these changes are what the developers have as defaults, so I decided to leave them in.

In helm when you run

helm show values spot/ocean-kubernetes-controller

It gives you what their default values file is for the version you are using. The changes you see in the values file are what their new suggested defaults are, and I kept a bit of our specific configuration as well that isn't the same as the default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got it. I saw it on the UI too:
Screenshot 2025-09-04 at 3 12 05 PM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we usually just copy the template that they provide? I can find template here: https://artifacthub.io/packages/helm/spot/ocean-kubernetes-controller

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly! I do it through helm commands, but there are a bunch of places that we can get their template.

Copying the default template and adjusting it is how I have done all of the helm based deployments in the repo

runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
runAsUser: 1000690000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on my understanding, both 10001 and 1000690000 are both non-root users. Why do we need to switch from one non-root user to another?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I undestand why now! For some reasons, that was changed in their template:
Screenshot 2025-09-04 at 3 11 01 PM

@spacelift-int-sagebionetworks spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-kubernetes-deployments September 5, 2025 16:14 Inactive
@BryanFauble BryanFauble changed the title Update ocean-kubernetes-controller version to 0.1.59 [DPE-1423] Update ocean-kubernetes-controller version to 0.1.59 Sep 5, 2025
@BryanFauble BryanFauble marked this pull request as ready for review September 5, 2025 16:23
@BryanFauble BryanFauble requested a review from a team as a code owner September 5, 2025 16:23
@spacelift-int-sagebionetworks spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-kubernetes-deployments September 5, 2025 16:39 Inactive
Copy link
Contributor

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 LGTM! Thanks for doing this update!

@BryanFauble BryanFauble merged commit 2d5025c into main Sep 8, 2025
6 of 7 checks passed
@BryanFauble BryanFauble deleted the upgrade-spot-helm-chart branch September 8, 2025 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants